Composite image creation system, composite image creation method, and non-transitory computer readable medium storing composite image creation program

ABSTRACT

A composite image creation system includes: a moving object region specifying unit that specifies a moving object region in each of a plurality of images consecutive in times series; a luminance correction unit that executes luminance correction processing to reduce a difference between a luminance signal in at least a part of one moving object region in one image and a luminance signal in at least a part of another moving object region in another image; and an image combining unit that combines the one image including the one moving object region subjected to the luminance correction processing and the other image including the other moving object region subjected to the luminance correction processing, and generates a composite image by adding predetermined color information to at least a part of the one moving object region and at least a part of the other moving object region in time series.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation from PCT application No.PCT/JP2015/000560, filed Feb. 6, 2015, which claims the benefit ofpriority from Japanese patent application No. 2014-067642, filed Mar.28, 2014, the disclosure of which is incorporated herein in its entiretyby reference.

BACKGROUND

The present invention relates to a composite image creation system, acomposite image creation method, and a non-transitory computer readablemedium storing a composite image creation program. In particular, thepresent invention relates to a composite image creation system, acomposite image creation method, and a non-transitory computer readablemedium storing a composite image creation program which are capable ofeasily grasping temporal information from a composite image.

There has been conventionally known an athletic motion analysissupporting device (for example, see Japanese Unexamined PatentApplication Publication No. 2002-298143) including: storage processingmeans for receiving images supplied from image pickup means and storingthe images in a temporary storage unit; means for reading out apredetermined number of images stored in the temporary storage unit andgenerating a reference image from a plurality of read images to beprocessed; trajectory image generation means for comparing the generatedreference image with each image to be processed and superimposing, onthe reference image, different areas in which each image to be processeddiffers by a predetermined amount from the reference image, therebygenerating a trajectory image; and trajectory image output means foroutputting the generated trajectory image to an output device. Accordingto the athletic motion analysis supporting device disclosed in JapaneseUnexamined Patent Application Publication No. 2002-298143, informationfor quantitatively and intuitively analyzing the motion to be monitoredof a facility or the like can be output by simple image processing,without depending on the situation outdoors or indoors.

SUMMARY

In the athletic motion analysis supporting device as disclosed inJapanese Unexamined Patent Application Publication No. 2002-298143, itis difficult to grasp a temporal relationship between a plurality ofobjects included in an image.

In order to solve the above-mentioned problem, an exemplary embodimentprovides a composite image creation system including: a moving objectregion specifying unit that specifies a moving object region including amoving object in each of a plurality of images consecutive in timesseries; a luminance correction unit that executes luminance correctionprocessing to reduce a difference between a luminance signal in at leasta part of one moving object region in one image and a luminance signalin at least a part of another moving object region in another image;

and an image combining unit that combines the one image including theone moving object region subjected to the luminance correctionprocessing with the other image including the other moving object regionsubjected to the luminance correction processing, and generates acomposite image by adding predetermined color information to at least apart of the one moving object region and at least a part of the othermoving object region in time series.

In order to solve the above-mentioned problem, the exemplary embodimentprovides a composite image creation method including: a moving objectregion specifying step of specifying a moving object region including amoving object in each of a plurality of images consecutive in timesseries; a luminance correction step of executing luminance correctionprocessing to reduce a difference between a luminance signal in at leasta part of one moving object region in one image and a luminance signalin at least a part of another moving object region in another image; andan image combining step of combining the one image including the onemoving object region subjected to the luminance correction processingwith the other image including the other moving object region subjectedto the luminance correction processing, and generating a composite imageby adding predetermined color information to at least a part of the onemoving object region and at least a part of the other moving objectregion in time series.

Furthermore, in order to solve the above-mentioned problem, theexemplary embodiment provides a non-transitory computer readable mediumstoring a composite image creation program for a composite imagecreation system, the composite image creation program causing a computerto implement: a moving object region specifying function for specifyinga moving object region including a moving object in each of a pluralityof images consecutive in times series; a luminance correction functionfor executing luminance correction processing to reduce a differencebetween a luminance signal in at least a part of one moving objectregion in one image and a luminance signal in at least a part of anothermoving object region in another image; and an image combining functionfor combining the one image including the one moving object subjected tothe luminance correction processing with the other image including theother moving object region subjected to the luminance correctionprocessing, and generating a composite image by adding predeterminedcolor information to at least a part of the one moving object region andat least a part of the other moving object region in time series.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a functional configuration of acomposite image creation system according to an exemplary embodiment;

FIG. 2 is a schematic diagram showing processing in a luminancecorrection unit according to the exemplary embodiment;

FIG. 3 is a process diagram showing the composite image creation systemaccording to the exemplary embodiment;

FIG. 4 is a diagram showing an image including a plurality of movingobjects when a system of a reference example is used instead of usingthe system of this exemplary embodiment;

FIG. 5 is a diagram showing an image including a plurality of movingobjects when the system of the reference example is used instead ofusing the system of this exemplary embodiment;

FIG. 6A is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6B is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6C is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6D is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6E is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6F is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6G is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 6H is a schematic diagram showing an image including a plurality ofmoving objects when the system of the reference example is used;

FIG. 7A is a schematic diagram schematically showing a composite imagewhen a luminance correction is not performed;

FIG. 7B is a schematic diagram schematically showing a composite imagewhen the luminance correction is not performed;

FIG. 7C is a schematic diagram schematically showing a composite imagewhen the luminance correction is not performed;

FIG. 8 is a detailed diagram showing processing in the composite imagecreation system according to the exemplary embodiment;

FIG. 9 is a detailed diagram showing processing in the composite imagecreation system according to the exemplary embodiment;

FIG. 10 is a graph showing how to change colors to be added to eachmoving object when 16 images are combined in the composite imagecreation system according to the exemplary embodiment;

FIG. 11 is a diagram showing how to change colors to be added to eachmoving object when 16 images are combined in the composite imagecreation system according to the exemplary embodiment;

FIG. 12 is a flowchart showing processing of the composite imagecreation system according to the exemplary embodiment; and

FIG. 13 is a hardware block diagram showing the composite image creationsystem according to the exemplary embodiment.

DETAILED DESCRIPTION

(Outline of Composite Image Creation System 1)

A composite image creation system 1 according to an exemplary embodimentis a system that generates a composite image including a trajectory ofeach of a plurality of objects having a motion (i.e., moving objects)included in a plurality of images constituting a moving image based onthe plurality of images, and generates a composite image with which atemporal relationship between moving objects can be easily grasped.According to the composite image creation system 1, a plurality ofmoving objects each having a motion in one composite image can bedistinguished and the temporal relationship between the moving objectscan be grasped at a glance.

(Details of Composite Image Creation System 1)

FIG. 1 shows an example of functional configuration blocks of thecomposite image creation system according to this embodiment.

The composite image creation system 1 includes an image storage unit 10that stores a moving image; a moving object region specifying unit 20that specifies a moving object region which is a region in which movingobjects included in the image are present; a luminance correction unit30 that performs luminance correction processing on at least a part ofthe moving object region within the image; an image combining unit 40that generates a composite image; and an output unit 50 that outputs thecomposite image. The moving object region specifying unit 20 includes animage comparison unit 200 that compares a plurality of images using oneof the plurality of images as a reference image; an integration unit 205that integrates binary signals based on a comparison result of the imagecomparison unit 200; and a key signal generation unit 210 that generatesa key signal based on an integration result of the integration unit 205.

(Image Storage Unit 10)

The image storage unit 10 stores a moving image. Specifically, the imagestorage unit 10 stores a moving image including moving objects. Morespecifically, the image storage unit 10 stores a plurality of imageframes which constitute the moving image and are consecutive in timesseries. The image storage unit 10 supplies the moving image to themoving object region specifying unit 20 and the luminance correctionunit 30 in response to actions performed by the moving object regionspecifying unit 20 and the luminance correction unit 30.

(Moving Object Region Specifying Unit 20)

The moving object region specifying unit 20 specifies a moving objectregion including a moving object in each of a plurality of images whichare stored in the image storage unit 10 and are consecutive in timesseries. Specifically, the moving object region specifying unit 20includes the image comparison unit 200 that compares one image withanother image which are stored in the image storage unit 10 andcalculates a difference between the one image and the other image; theintegration unit 205 that integrates calculation results of the imagecomparison unit 200; and the key signal generation unit 210 thatgenerates a key signal for specifying a moving object region included inthe one image based on the integration result of the integration unit205.

More specifically, the image comparison unit 200 acquires one image(hereinafter referred to as a “first image”) and another image(hereinafter referred to as a “second image”) from the images stored inthe image storage unit 10. The image comparison unit 200 calculates adifference between the luminance value or hue value of the first imageand the luminance value or hue value of the second image for each pixelor each pixel block. When the absolute value of the calculateddifference is greater than a predetermined first threshold, the imagecomparison unit 200 generates a binary signal indicating “1”, and whenthe absolute value is equal to or smaller than the first threshold, theimage comparison unit 200 generates a binary signal indicating “0”. Theimage comparison unit 200 supplies the generated binary signal to theintegration unit 205.

Further, the image comparison unit 200 acquires an image (hereinafterreferred to as a “third image”), which is different from the first andsecond images, from the image storage unit 10. The image comparison unit200 calculates a difference between the first image and the third imageand generates a binary signal in the same manner as that describedabove. The image comparison unit 200 supplies the generated binarysignal to the integration unit 205.

The integration unit 205 adds up the binary signals received from theimage comparison unit 200. Specifically, the integration unit 205 addsup the binary signal calculated based on the first and second images andthe binary signal calculated based on the first and third images.

The image comparison unit 200 compares the first image with stillanother image (hereinafter referred to as a “fourth image”) andcalculates the difference therebetween in the same manner as thatdescribed above, and generates a binary signal in the same manner asthat described above. Further, the integration unit 205 adds up thebinary signals obtained by integrating and the binary signals calculatedbased on the first image and the fourth image.

In this manner, when the image comparison unit 200 acquires N (N is aninteger equal to or greater than 2) images to be compared from the imagestorage unit 10, the image comparison unit 200 calculates a plurality ofbinary signals based on the difference between the first image and eachof the images ranging from the second image to the N-th image. Further,the integration unit 205 integrates the plurality of binary signalscalculated by the image comparison unit 200, and generates an integratedsignal of the binary signals with a signal level in a range from 0 toN−1. The integration unit 205 supplies the generated integrated signalto the key signal generation unit 210.

The key signal generation unit 210 controls the amplitude of the signallevel of the integrated signal to fall within a range of predeterminedconstant values by using a predetermined second threshold for theintegrated signal supplied from the integration unit 205. Accordingly,the key signal generation unit 210 generates a first key signal which isused to extract a moving object region including moving objects of thefirst image.

Next, the image comparison unit 200 and the integration unit 205 executethe same processing as that described above on the second image and N−1images excluding the second image from the images ranging from the firstimage to the N-th image. Further, the key signal generation unit 210generates a second key signal in the same manner as that describedabove. The moving object region specifying unit 20 executes the sameprocessing as that described above on each of the images ranging fromthe third image to the N-th image, thereby generating third to N-th keysignals. The key signal generation unit 210 supplies the generated keysignals to the luminance correction unit 30.

Note that the image comparison unit 200 treats the N−1 images excludingan image to be compared from the images ranging from the first image tothe N-th image. However, in the comparison between the same images (forexample, in the comparison between the first image and the first image),the difference between the images is “0”. In this case, the binarysignal is fixed to “0”. Accordingly, in order to simplify a controlcircuit, the image comparison unit 200 can perform comparison processingon each of the N images from the first image to the N-th image.

(Luminance Correction Unit 30)

The luminance correction unit 30 executes luminance correctionprocessing to reduce a difference between a luminance signal in at leasta part of one moving object region in one image and a luminance signalin at least a part of another moving object region in another image.Specifically, the luminance correction unit 30 acquires the first imagefrom the image storage unit 10. Further, the luminance correction unit30 performs luminance correction processing on the region specified bythe first key signal based on coordinate information of the first keysignal generated by the key signal generation unit 210. Note that theluminance correction unit 30 can perform luminance correction processingonly on a region in the vicinity of the contour of the region specifiedby the first key signal. The luminance correction unit 30 supplies thefirst image subjected to luminance correction processing to the imagecombining unit 40.

Next, the luminance correction unit 30 acquires the second image fromthe image storage unit 10. Further, the luminance correction unit 30performs luminance correction processing on the region specified by thesecond key signal based on coordinate information of the second keysignal generated by the key signal generation unit 210. The luminancecorrection unit 30 supplies the second image subjected to luminancecorrection processing to the image combining unit 40.

(Image Combining Unit 40)

The image combining unit 40 overwrites only the moving object regionfrom a plurality of images, thereby generating a composite imageincluding a trajectory of each moving object. Specifically, the imagecombining unit 40 combines one image including one moving object regionsubjected to luminance correction processing with another imageincluding another moving object region subjected to luminance correctionprocessing, and generates a composite image by adding predeterminedcolor information to at least a part of one moving object region and atleast a part of another moving object region in time series.

For example, the image combining unit 40 combines the first image andthe second image which are subjected to luminance correction processingby using the amplitude information of the second key signal obtained asdescribed above. When the key signal is a signal having an amplitudefrom 0 to m, the image combining unit 40 generates a composite imagebased on a formula of (first image subjected to luminancecorrection)×(m−second key signal)/m+(second image subjected to luminancecorrection)×(second key signal)/m.

The image combining unit 40 uses, as it is, the first image subjected tothe luminance correction processing for the portion in which the secondkey signal indicates “0”, and uses, as it is, the second image subjectedto the luminance correction processing for the portion in which thesecond key signal indicates “m”. Further, the image combining unit 40generates an image in which the first image and the second image thatare subjected to the luminance correction processing are mixed at aratio corresponding to the value of the second key signal in the portionin which the second key signal indicates a value between 0 and m. Theimage combining unit 40 supplies, to the image storage unit 10, thecomposite signal of the image in which the first image and the secondimage are mixed. The image storage unit 10 stores the composite signalas a composite image (1+2).

Next, the luminance correction unit 30 acquires the third image from theimage storage unit 10, executes the luminance correction processing inthe same manner as that described above, and supplies the third imagesubjected to the luminance correction processing to the image combiningunit 40. The image combining unit 40 generates a composite signal usingthe third image subjected to the luminance correction processing, thecomposite image (1+2) stored in the image storage unit 10, and the thirdkey component signal in the same manner as that described above when thecomposite image (1+2) is generated. The image combining unit 40 suppliesthe generated composite signal to the image storage unit 10. The imagestorage unit 10 stores the composite signal as a composite image(1+2+3).

The luminance correction unit 30 and the image combining unit 40 performthe above-described processing on each of the images ranging from thefirst image to the N-th image. Thus, a composite image (1+2+ . . . N) isstored in the image storage unit 10. The composite image (1+2+ . . . N)is the composite image finally obtained. Note that the luminancecorrection unit 30 can generate a composite image by using, as it is, acolor signal of each image as a color signal specified by each keysignal, without executing the luminance correction processing.

The image combining unit 40 can generate a composite image usingpredetermined color information in time series as color informationabout the moving object region specified by each key signal. In thiscase, the image combining unit 40 performs the luminance correctionprocessing on the moving object region, and adds the color informationto the entire moving object region. The image combining unit 40 can alsoadd the color information to the contour of one moving object region andthe contour of another moving object region. In this case, the imagecombining unit 40 performs the luminance correction processing on thecontour region of each moving object and adds color information only tothe contour region.

When the color information is added to each moving object, the imagecombining unit 40 can determine the color information to be added basedon the background color of one moving object region and the backgroundcolor of another moving object region. For example, when images includean image of a game in a place, such as on a lawn, the background colorof the image is a color of a green color system, and when images includean image of a game in a place, such as on the ground, the backgroundcolor of the image is a color of a yellow or brown color system. In thiscase, if the color information to be added to each moving object of thecomposite image indicates a color of the same color system as thebackground colors, it is difficult to visually recognize the movingobject. Accordingly, the image combining unit 40 can prevent a color ofthe same color system as the background color of the image from beingadded to each moving object during generation of the composite image.

Further, when the color information is added, the image combining unit40 can use a monochrome image as the background color of one movingobject region and as the background color of another moving objectregion. For example, when various colors are present in the backgroundof an image and the colors affect the visibility of a color representingtemporal information of a composite image, the image combining unit 40can use a monochrome image as the background image.

The image combining unit 40 can also add the color information to atleast a part of one moving object region and at least a part of anothermoving object region based on a predetermined wavelength order.Specifically, the image combining unit 40 can generate the trajectory ofrainbow colors by adding, for the trajectory of each moving object, thecolor information to at least a part of the moving object region indescending order of wavelength or ascending order of wavelength in timeseries. The image combining unit 40 supplies the generated compositeimage to the output unit 50.

(Output Unit 50)

The output unit 50 visibly outputs the composite image to a user. Theoutput unit 50 is, for example, a display device such as a monitor. Theoutput unit 50 may be an output device such as a printer for printingthe composite image. Further, the output unit 50 can output data of thecomposite image to the outside of the composite image creation system 1.

(Outline of Luminance Correction Processing)

FIG. 2 shows an outline of processing in the luminance correction unitaccording to this embodiment.

The luminance correction unit 30 executes the luminance correctionprocessing to set luminance signals of a plurality of moving objectscloser to each other, without specifying the exact moving object regionof each moving object present in one image. For example, the luminancecorrection unit 30 executes the luminance correction processing based ona nonlinear circuit.

This processing will be described in detail with reference to FIG. 2.FIG. 2 is a graph showing y=x where y represents a linear output withrespect to an input signal (horizontal axis), and also showingy′=x^(0.2) where y′ represents a nonlinear output with respect to theinput signal. For example, assume that average luminances of two movingobjects whose luminance signals need to be set closer to each other are“0.2” and “0.6”, respectively. In this case, the ratios between theluminance levels of the two moving objects differ by a factor of 3. Whenthe signals are supplied to a circuit having a function of y′=x^(0.2),the signal representing the average luminance of “0.2” indicates “0.72”and the signal representing the average luminance of “0.6” indicates“0.90”. Thus, the ratio between the signals is 1:1.25. Although it isdifficult to completely match the luminance levels of the two movingobjects, the approximation of the ratios between the luminance signalsmakes it possible to more easily compare the images colored withdifferent colors, than when the luminance signals before conversion areused. Since a signal in a region with a low luminance level is convertedinto a signal with a high luminance level as shown in FIG. 2, thegeneration of an entirely dark composite image in which the luminancesignal is low and it is difficult to distinguish the colors can beprevented. Note that the luminance correction unit 30 may calculateaverage luminances of moving object regions specified by a plurality ofkey signals present in one image, and may execute the luminancecorrection processing based on the calculated luminances.

(Outline of Contour Extraction)

FIG. 3 shows an example of processing in the composite image creationsystem according to this embodiment.

The luminance correction unit 30 can extract the contour of each movingobject by using, for example, a high-pass filter (HPF) for extracting ahigh-frequency component of the key signal of each moving object. Theluminance correction unit 30 can also extract the contour of each movingobject in the manner as described below.

First, the key signal generation unit 210 generates each key signalbased on the integrated value of binary signals representing differencesobtained by comparing a plurality of images. A case will be described inwhich, as shown in the images on the left side of FIG. 3, the number ofimages is five and the key signal of “IMAGE 3”, which is a middle pointof the movement of the five images when the moving object is graduallymoving rightward, is generated.

When the “IMAGE 3” and “IMAGE 1” shown in FIG. 3 are compared, theluminance correction unit 30 determines that a difference equal to orgreater than a set value of a first threshold is generated and executesprocessing on the region excluding the background image which is commonto “IMAGE 1” and “IMAGE 3”. Similarly, the luminance correction unit 30executes processing on “IMAGE 3” and “IMAGE 2”, “IMAGE 3” and “IMAGE 4”,and, “IMAGE 3” and “IMAGE 5”. In this case, binary signals as shown inthe upper right of FIG. 3 are obtained.

When the integration unit 205 integrates the four binary signals shownin FIG. 3, signals having signal levels of 0 to 4 as shown in the centeron the right side of FIG. 3 are obtained.

When values equal to or less than the level 1 of the signals are cut bythe second threshold in the key signal generation unit 210, the keysignal as shown in the lower right of FIG. 3 is created. In this case,when the portion corresponding to the level 3 of the key signal iscontrolled as a moving object portion of “IMAGE 3” and the portioncorresponding to the level 1 of the key signal is controlled as acontour signal of “IMAGE 3”, the luminance correction unit 30 canextract the contour of “IMAGE 3” without the need for adding a contourdetection processing function for each key signal.

Note that in the example shown in FIG. 3, since “IMAGE 3” at the centertime of movement of the images to be used is used, the contour signalthat is uniform on the right and left sides is obtained, while in “IMAGE1” and “IMAGE 5”, the contour signal that is biased to one side isobtained. However, the luminance correction unit 30 can employ theabove-described method to reduce the processing load in the system byemploying simple contour extraction processing.

(Processing in Reference Example)

FIGS. 4 and 5 show examples of an image including a plurality of movingobjects when a system of a reference example is used instead of usingthe system of this embodiment. FIGS. 6A to 6H are examples of an outlineschematically showing an image including a plurality of moving objectswhen the system of the reference example is used. FIGS. 7A to 7C showexamples of an outline schematically showing a composite image when theluminance correction is not performed.

First, as shown in FIGS. 4 and 5, a case where a plurality of movingobjects are present in an image (FIGS. 4 and 5 show a scene in which,for example, two racing cars turn a corner) will be described. In thisexample, as shown in FIG. 4, a vehicle with a low luminance levelprecedes a vehicle with a high luminance level. FIG. 5 shows an imageobtained by combining trajectories by using images preceding andfollowing the image shown in FIG. 4. Referring to FIG. 5, it can beunderstood that two vehicles take different routes, but it is difficultto understand which one of the vehicles precedes the other, or it isdifficult to grasp the location of the vehicle traveling behind in theimage when the preceding vehicle is located at a predetermined position.This state is schematically shown in FIGS. 6A to 6H.

FIGS. 6A to 6F show frame images which vary with time. At the time shownin FIG. 6A, an object A which has a trapezoidal shape, has a lowluminance signal level, and is represented by a checkered patternappears on the left side of the screen. At this time, a circular, whiteobject B which has a high luminance level and is represented by a dashedline has not appeared on the screen yet. At the time shown in FIG. 6B,the object B enters into the screen from the left side of the screen,and the object A moves rightward. The objects A and B move rightwardwith time to the states as shown in FIGS. 6C to 6E, and at the timeshown in FIG. 6F, the object A moves outside of the screen from theright side of the screen as represented by a dashed line. When thetrajectories of these images are combined, an image shown in FIG. 6G canbe obtained. Five images of the object A, as well as five images of theobject B are seen in the screen, and the traveling speed of the object Ais higher than the traveling speed of the object B. In this case, forexample, when racing cars turn a corner as mentioned above, not only thedifference in speed between the cars, but also trajectories, such asroute taking required for cornering, can be confirmed. However, inpractice, the temporal contexts of the objects A and B have a relationas represented by arrows in FIG. 6H. Accordingly, when the systemaccording to this embodiment is not used, it is difficult to grasp thetemporal positional relationship between the object A and the object Bas shown in FIG. 6G.

A case where colors are added to the respective moving objects as shownin FIGS. 7A to 7C will now be considered. In the examples shown in FIGS.7A to 7C, the colors are represented by patterns such as oblique lines.FIGS. 7A and 7C each show a state in which the color signal varies withtime in such a manner that “plain→oblique lines slanting to theright→vertical lines→oblique lines slanting to the left→horizontallines→dotted pattern”. In this case, a composite image obtained bycombining the trajectories of the moving objects as shown in FIG. 7A isobtained. In FIG. 7A, C indicates an index display showing time. In thiscase, the relation between the index display C and the object B which isplain and has a high signal level is extremely easy to grasp. However,it is difficult to grasp the relation between the index display C andthe object A which has some patterns and a low luminance level. As aresult, it is difficult to grasp the temporal relationship between theobject A and the object B. If the number of types of color codes issmall, distinguishable colors, such as “red→yellow→green→blue→violet”,can be selected even when the luminance levels and patterns of themoving objects are slightly different. However, as the number of typesof color codes increases, more colors such as intermediate colors areused. In this case, the influence of the luminance levels and patternsof the moving objects increases. As a result, it becomes difficult todistinguish the temporal relationship between the objects.

FIG. 7B shows a case where the density of the color to be added to eachmoving object is changed. Also in this case, it is difficult to graspthe temporal relationship between the objects A and B and the indexdisplay C depending on the original luminance level, color, and/orpatterns of the object A. FIG. 7C shows a case where the patterns to beadded to each moving object is changed. In this case, since attention isfocused only on the patterns, the temporal relationship between theobjects A and B and the index display C is clear. However, the originalpatterns or the like of each moving object are erased. In the case ofFIG. 7C, the two moving objects have distinctly different shapes, suchas a trapezoid and a circle, which makes it possible to distinguish thetwo moving objects. However, many of the images to be used for creatinga composite image by combining trajectories of a plurality of movingobjects are, for example, images of racing cars and athletes playing agame. Thus, the shapes of moving objects to be distinguished in oneimage are extremely similar to each other. Therefore, the discriminationof moving objects depends on, for example, the patterns of vehicles, thecolor or patterns of uniforms, or the facial expressions of people.Accordingly, if the original features of each moving object aredrastically changed, it becomes difficult to distinguish the pluralityof moving objects.

(Details of Processing in the Composite Image Creation System 1)

FIGS. 8 and 9 schematically show the details of processing in thecomposite image creation system according to this embodiment.

As described above with reference to the functional configuration blockdiagram, the composite image creation system 1 according to thisembodiment executes signal processing capable of grasping temporal axisinformation, while enabling to discriminate a plurality of movingobjects. First, the composite image creation system 1 adds temporal axisinformation based on an order of colors to the composite image aftersetting the luminance signal levels of the plurality of moving objectscloser to each other. FIG. 8 is a diagram showing a state in whichprocessing for setting the luminance levels of the objects A and Bcloser to each other within a range in which the patterns of the objectA are recognizable as a moving object is performed on FIG. 7A.

In this figure, the colors are represented by patterns such as obliquelines in the same manner as shown in FIGS. 7A and 7C. The object A andthe object B have similar luminance levels and the same color, but havedifferent patterns. This makes it possible to distinguish the twoobjects and easily grasp the temporal relationship between the objects.In this case, if the luminance level is lowered, each color becomesdarker, and if the luminance level is extremely high, white-out occursin each color. Accordingly, the luminance correction unit 30 preferablygenerates the luminance signal at an intermediate level.

Further, the composite image creation system 1 can add temporal axisinformation based on fixed luminance levels and an order of colors onlyto the contour of each object. An example of this case is shown in FIG.9. The discrimination of each moving object is executed using a portionother than the contour of each moving object, and the temporalrelationship is grasped using the color of the contour portion.

Although not shown, the composite image creation system 1 can executethe following methods in combination with the above-described method.That is, the composite image creation system 1 can execute, for example,a method for coloring the entire surface of a plurality of movingobjects and setting the luminance level of only the contour portion ofeach moving object to a fixed value; a method for coloring the entiresurface of signals in which the luminance levels of a plurality ofmoving objects are set closer to each other and setting the luminancelevel of the contour of each moving object to a fixed value; or a methodfor setting the luminance levels of moving objects closer to each otherin a portion of each moving object other than the contour thereof.Furthermore, the composite image creation system 1 can add a markerhaving a predetermined shape to a part of each moving object included inan image.

(Method for Adding Color Information)

FIGS. 10 and 11 show examples of how to change the color to be added toeach moving object when 16 images are combined in the composite imagecreation system according to this embodiment.

When the image combining unit 40 adds color information to each movingobject, a circuit or program generates a fixed color by providing apredetermined function to two color difference signals R-Y and B-Y or byusing data on color information read out of a prepared database. Asshown in FIG. 10, for example, when the image combining unit 40 combines16 images (indicated by 0 to 15 in FIG. 10), in the color information tobe added to moving objects included in each image, R-Y components arechanged by Cos function according to the order of composition, and B-Ycomponents are changed by -Sin function according to the order ofcomposition. In this case, the color signal starts from a point A shownin FIG. 11, and the color of the color signal is changedcounterclockwise and the color change ends at a point B. However, if thepoint A which represents a first color is close to the point B whichrepresents a last color, it may be difficult to distinguish the startpoint from the end point. For this reason, the image combining unit 40use colors with different brightness or chroma levels for colorinformation to be added to a first moving object region at a start timein time series and for color information to be added to a second movingobject region at an end time in the time series. Specifically, the imagecombining unit 40 determines the order of colors for 16 divided regionsstarting from red as shown in FIG. 11, and the color is changedcounterclockwise and the color change ends at a point of violet. Then,the image combining unit 40 adds the determined color information toeach moving object.

(Composite Image Creation Method)

FIG. 12 shows an example of a processing flow of the composite imagecreation system according to the exemplary embodiment.

First, the moving object region specifying unit 20 specifies the movingobject region which is included in each of a plurality of images storedin the image storage unit 10 and indicates the location of each movingobject in the plurality of images (step 10; the step is represented as“S”). Next, the luminance correction unit 30 executes the luminancecorrection processing to reduce the difference between the luminancesignal in at least a part of the moving object region of one referenceimage and the luminance signal in at least a part of the moving objectregion of another image (S15). Subsequently, the image combining unit 40combines the plurality of images including the moving object regionsubjected to the luminance correction processing, and generates acomposite image by adding predetermined color information to at least apart of one moving object region and at least a part of another movingobject region in time series (S20). Then, the output unit 50 outputs thecomposite image (S25).

FIG. 13 shows an example of a hardware configuration of the compositeimage creation system according to the exemplary embodiment.

The composite image creation system 1 according to this exemplaryembodiment includes: a CPU 1500; a graphic controller 1520; a memory1530 such as a Random Access Memory (RAM), a Read-Only Memory (ROM),and/or a flash ROM; a storage device 1540 that stores data; a read/writedevice 1545 that reads data from a recording medium and/or writes datainto a recording medium; an input device 1560 that inputs data; acommunication interface 1550 that transmits data to an externalcommunication device and receives data therefrom; and a chip set 1510that connects the CPU 1500, the graphic controller 1520, the memory1530, the storage device 1540, the read/write device 1545, the inputdevice 1560, and the communication interface 1550 to each other in sucha manner that they can communicate with each other.

The chip set 1510 connects the memory 1530, the CPU 1500, which accessesthe memory 1530 and executes predetermined processing, and the graphiccontroller 1520, which controls the display of an external displaydevice, to each other, thereby executing data delivery between thecomponents. The CPU 1500 operates based on a program stored in thememory 1530 and controls each component. The graphic controller 1520causes a predetermined display device to display images based on imagedata temporarily stored in a buffer which is provided in the memory1530.

Further, the chip set 1510 connects the storage device 1540, theread/write device 1545, and the communication interface 1550 to eachother. The storage device 1540 stores a program and data to be used bythe CPU 1500 of the composite image creation system 1. The storagedevice 1540 is, for example, a flash memory. The read/write device 1545reads a program and/or data from the storage medium storing the programand/or data, and stores the read program and/or data in the storagedevice 1540. The read/write device 1545 acquires a predetermined programfrom a server on the Internet via, for example, the communicationinterface 1550, and stores the acquired program in the storage device1540.

The communication interface 1550 executes transmission and reception ofdata with an external device via a communication network. When thecommunication network is disconnected, the communication interface 1550can execute transmission and reception of data with the external devicewithout involving the communication network. The input device 1560, suchas a keyboard, a tablet, or a mouse, is connected to the chip set 1510via a predetermined interface.

A composite image creation program for the composite image creationsystem 1 that is stored in the storage device 1540 is provided to thestorage device 1540 via a communication network, such as the Internet,or via a recording medium, such as a magnetic recording medium or anoptical recording medium. The composite image creation program for thecomposite image creation system 1 that is stored in the storage device1540 is executed by the CPU 1500.

The composite image creation program executed by the composite imagecreation system 1 according to this embodiment works on the CPU 1500 tocause the composite image creation system 1 to function as the imagestorage unit 10, the moving object region specifying unit 20, theluminance correction unit 30, the image combining unit 40, the outputunit 50, the image comparison unit 200, the integration unit 205, andthe key signal generation unit 210 which are described above withreference to FIGS. 1 to 12.

(Effects Of Embodiments)

The composite image creation system 1 according to this embodimentexecutes the processing as described above in the embodiments, therebyenabling discrimination of moving objects as a plurality of movingobjects in one trajectory composite image and generation of thecomposite image with which the relationship between the temporal axes ofthe plurality of moving objects can be easily recognized. In otherwords, the composite image creation system 1 can generate the compositeimage with which a plurality of moving objects each having a motion canbe distinguished and the temporal relationship between the movingobjects can be grasped at a glance.

While the exemplary embodiments have been described above, the inventionaccording to the claims is not limited by the exemplary embodimentsdescribed above. It should be noted that not all the combinations of thefeatures described in the exemplary embodiments are essential as meansfor solving the problems of the invention. Further, the technicalelements in the embodiments described above may be applied singly, ormay be applied by dividing them into a plurality of portions such asprogram components and hardware components.

What is claimed is:
 1. A composite image creation system comprising: amoving object region specifying unit configured to specify a movingobject region including a moving object in each of a plurality of imagesconsecutive in times series; a luminance correction unit configured toexecute luminance correction processing to reduce a difference between aluminance signal in at least a part of one moving object region in oneimage and a luminance signal in at least a part of another moving objectregion in another image; and an image combining unit configured tocombine the one image including the one moving object region subjectedto the luminance correction processing with the other image includingthe other moving object region subjected to the luminance correctionprocessing, and generate a composite image by adding predetermined colorinformation to (i) at least a part of the one moving object region and(ii) at least a part of the other moving object region, thepredetermined color information comprising a predetermined order ofcolor information that changes in time series.
 2. The composite imagecreation system according to claim 1, wherein the image combining unitadds the color information to a contour of the one moving object regionand a contour of the other moving object region.
 3. The composite imagecreation system according to claim 1, wherein when the color informationis added, the image combining unit determines the color information tobe added based on a background color of the one moving object region anda background color of the other moving object region.
 4. The compositeimage creation system according to claim 1, wherein when the colorinformation is added, the image combining unit uses a monochrome imageas a background color of the one moving object region and as abackground color of the other moving object region.
 5. The compositeimage creation system according to claim 1, wherein the image combiningunit adds the color information to at least a part of the one movingobject region and at least a part of the other moving object regionbased on a predetermined wavelength order.
 6. The composite imagecreation system according to claim 5, wherein when the image combiningunit generates the composite image including a plurality of movingobject regions in the time series and adds a number of pieces of thecolor information corresponding to a predetermined number of colors tothe plurality of moving object regions, respectively, the imagecombining unit uses colors with different brightness or chroma levelsfor color information to be added to a first moving object region at astart time in the time series and for color information to be added to asecond moving object region at an end time in the time series.
 7. Acomposite image creation method comprising: a moving object regionspecifying step of specifying a moving object region including a movingobject in each of a plurality of images consecutive in times series; aluminance correction step of executing luminance correction processingto reduce a difference between a luminance signal in at least a part ofone moving object region in one image and a luminance signal in at leasta part of another moving object region in another image; and an imagecombining step of combining the one image including the one movingobject region subjected to the luminance correction processing with theother image including the other moving object region subjected to theluminance correction processing, and generating a composite image byadding predetermined color information to (i) at least a part of the onemoving object region and (ii) at least a part of the other moving objectregion, the predetermined color information comprising a predeterminedorder of color information that changes in time series.
 8. Anon-transitory computer readable medium storing a composite imagecreation program for a composite image creation system, the compositeimage creation program causing a computer to implement: a moving objectregion specifying function for specifying a moving object regionincluding a moving object in each of a plurality of images consecutivein times series; a luminance correction function for executing luminancecorrection processing to reduce a difference between a luminance signalin at least a part of one moving object region in one image and aluminance signal in at least a part of another moving object region inanother image; and an image combining function for combining the oneimage including the one moving object subjected to the luminancecorrection processing with the other image including the other movingobject region subjected to the luminance correction processing, andgenerating a composite image by adding predetermined color informationto (i) at least a part of the one moving object region and (ii) at leasta part of the other moving object region, the predetermined colorinformation comprising a predetermined order of color information thatchanges in time series.